The volume of Internet traffic generated by large business corporations requires considerable computing resources to monitor and detect known security threats, in addition to detecting new and less apparent security threats. In security, reports of malicious activity from the past are frequently received but modern archiving conventions do not support a substantial backlog of data for confirming whether or not a company was previously affected by the identified threat. In particular, storing data for a lengthy period of time (e.g., longer than a few weeks for packet capture data, or 3 months for system log) is very costly.
Current solutions attempt to address these issues by stripping context from the log to reduce its data size. However, the current methods only allow for reducing the size of the archived log to about 25-50% of its original size which may still require considerable resources for long-term storage. Therefore, it may be desirable to have a system and method that addresses at least some of these issues, and improves upon existing practices.